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caricatures: On the psychological consistency of computer face recognition. 2nd Face 
and Gesture, pages 4-10, 1996.] have suggested that using a shape-free coding 
provides a ready means of doing this, at least the when the range of pose-angle is 
relatively small, perhaps ± 20° [T. Poggio and D. Beymer. Learning networks for 
face analysis and synthesis. Face and Gesture, pages 160-165, 1995.]. In this 
embodiment of the invention, the correspondence problem between faces is first 
solved by finding a pre-selected set of distinctive points (corners of eyes or mouths, 
for example) which are present in all faces. This is typically performed by hand 
during training. Those pixels thus defined as being part of the face can be warped to a 
standard shape by standard grey-level interpolation techniques, ensuring that the 
image-wise and face-wise co-ordinates of a given image are equivalent. If a rigid 
transformation to remove scale, location and orientation effects is performed on the 
point-locations, they can then be treated in the same way as the grey- levels, as again 
identical values for corresponding points on different faces will have the same 
meaning. 

Although these operations linearise the space, allowing interpolation between 
pairs of faces, they do not give an estimate of the dimensions. Thus, the acceptability 
as a face of an object cannot be measured; this reduces recognition [N. P. Costen, I. 
G. Craw, G. J. Robertson, and S. Akamatsu. Automatic face recognition: What 
representation ? European Conference on Computer Vision, vol 1, pages 504-513, 
1996]. In addition, redundancies between feature-point location and grey-level values 
cannot be described. Both these problems are addressed in this embodiment of the 
invention by Principal Components Analysis (PCA). This extracts a set of orthogonal 
eigenvectors O from the covariance matrix of the images (either the pixel grey- 
levels, or the featurepoint locations). Combined with the eigenvalues, this provides 
an estimate of the dimensions and range of the face- space. The weights w of a face q 
can then be found, 



w = <t> T (q -q) 



(1) 



and this gives the Mahalanobis distance 




(2) 
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spaces. This process is iterated to obtain a set of stable, and rather more orthogonal, 
sub-spaces which code only the desired features. 

If n s subspaces are used, each described by eigenvectors with the 
associated eigenvalues A, a) for a given flu q the projection out of the combined sub- 
spaces is given by: 

9 ' = £^>hP>+4 (4) 
with the constraints that 

j=l 1=1 >t. 

be minimised. Thus if M is the matrix formed by concatenating ^ (j=ia - } and D is the 
diagonal matrix of X 0=U2 } ' 

w = (DM T M + 7)" 1 DM T (q — q) (6) 
and this also gives a projected version of the face 

q =(DM T y\DM T M +I)w+q (7) 
with w. ~ 0 for those sub-spaces not required. 

The first stage of implementing the invention was to subtract the overall mean 
from each face, so ensuring that the mean of each sub-space was as close to zero as 
possible. Separate principle component analyses (PCAs) were then performed upon 
the image sets, discarding any further difference between the group and overall 
means. The covariance matrices for the identity and lighting sub-spaces were 
calculated as _ 

C T (<7,~?) r (8) 



n 



i-i 



the pose and expression used 

C »=— — £ Z -?,) r (9) 

n o U p 1=1 A=l 

where n 0 is the number of observations per individual, and n p is the number of 
individuals, and q i the mean of individual i. Although all the eigenvectors implied by 
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Convergence of the method was estimated by taking the Mahalanobis 
distances between all the images on each of the sub-spaces. A Pearson product- 
moment correlation was taken between the distances of successive iterations, and 
allowed to converge to machine accuracy, although in practice a slightly lower value 
would achieve the same results with reduced processing time. The method gave a 
relatively smooth set of correlation coefficients as shown in Figure 5, converging in 
approximately seven iterations (Figure 5 shows changes in the correlations between 
the Mahalanobis distances separating all the images on the multiple space between 
iteration n and nA). Since only 99.99% of the variance in the ensemble to avoid 
problems with numerical accuracy, practical convergence was achieved by the fourth 
iteration. 

Since the iterations involved the inclusion of information which failed to be 
coded on the previous iteration, it should be expected that the difference between 
original and projected images should decline. This should apply to both ensemble and 
non-ensemble images as the eigenfaces become more representative. 

This was tested by projecting the images through the combined spaces 
(using Equations 6 and 7) and measuring the magnitude of the errors. This was 
performed for both the ensemble images and also for a large test set (referred to as 
* Manchester 1 ), first used in [A. Lanitis, C. J. Taylor, and T. F. Cootes. An automatic 
face identification system using flexible appearance models. British Machine Vision 
Conference, pages 65-74, 1994]. This consisted 600 images of 30 individuals, 
divided in half: a gallery of 10 images per person and a set of 10 probes per person. 
As can be seen in Figure 6, in both cases, the errors quickly dropped to a negligible 
level (Errors quickly decline to a negligible level in both cases. Errors on the 
individual sub-spaces remain high (4,000 to 1 1,000)). As a comparison, the two sets 
have mean magnitudes (total variance) of 11345 and 11807, measured on the 
appearance-model eigenweights. 

The level of normalisation was measured on the Manchester set, calculating 
the identity weights using Equation 6, and finding the person-mean Wi . Better 
removal of contaminating variance should reduce the variance for each individual, 
relative to this mean. The variance, 
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1 




(12) 



,*=* k=) j=l 



was calculated. The results of this test in Figure 7 show a steady decline in the 
identity sub-space variance (Figure 7 shows the mean within-person variances for the 
different sub-spaces as a function of iteration number). The only exception to this is 
the value for iteration two; this is unusual in having a large increase in the number of 
dimensions, without an opportunity to re-distribute this variation into the other sub- 
spaces. 

The results of projecting the faces into the other sub-spaces are shown, as is 
the variance in the appearance model. As might be expected, these are all higher than 
the identity sub-space value, and do not show marked declines as the iterations 
progress. Indeed, the pose variance increases slightly. 

Recognition was also tested on the Manchester set, coding the images on the 
final rotated space. The Appearance Model used to provide correspondences, did not 
give completely accurate positions, lowering recogntion. The pooled covariance 
matrix was found using Equation 9 on the w,-. This allowed 



where 1 < k < (n 0 xn p ) to give Mahalanobis distances to the mean images. A 

recognition was scored when the smallest d had the same identity for / and k. The 
results are shown in Figure 8 (which shows recognition rates for Euclidean average- 
image matching), and demonstrate that relative to the base condition, recognition 
improves by about one percent on iteration 4. Also shown are the effects of 
projecting the test images through the complete space to obtain the lighting - pose - 
expression normalised version, and then coded on the final rotated space. This does 
not produce an improvement in recognition. It should be noted here that there may 
well be contingent, non-functional correlations between parameters on different sub- 
spaces for individuals (for example, a consistent tendency to look up or down), whose 
omission may trade off against theoretically preferable eigenfaces. 

Once an accurate coding system for faces has been achieved, the major 
problem is to ensure that only a useful sub-set of the codes are used for any given 
manipulation or measurement. This is a notably difficult task, as there are multiple, 



(13) 
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non-orthogonal explanations of any given facial configuration. In addition, it is 
typically the case that only a relatively small portion of the very large data-base 
required will be present in the full range of conditions and with the labels needed a 
simple linear extraction. 

The invention overcomes these problems by using an iterative receding 
scheme, which takes into account both the variance of and covariance between the 
sub-spaces which can be extracted to span sets of faces which vary in different ways. 
This yields 'cleaner' eigenfaces, with lower within appropriate group variance and 
higher inappropriate group variance. Both these facts reflect greater orthogonality 
between the sub-spaces. In addition, recognition on an entirely disjoint test set was 
improved, although marginally. The invention may be applied to tracking, lip-reading 
and transfer of identity from one person to another. 



SUBSTITUTE SHEET (RULE 26) 



WO 00/33240 



• 



13 



PCT/GB99/03953 



CLAIMS 

1. A method of determining face sub-spaces, the method comprising making 
initial estimates of the sub-spaces, for example lighting, pose, identity and expression, 
using Principle Component Analysis on appropriate groups of faces, applying an 
iterative algorithm to image codings to maximise the probability of coding across 
these non-orthogonal sub-spaces, obtaining the projection on each sub-space, and 
recalculating the spaces. 

2. A method of determining face sub-spaces, the method comprising: 

a. generating a first series of initial images in which a first predetermined facial 
property is modified, 

b. generating a second series of initial images in which a second predetermined 
facial property is modified, 

c. coding each series of images according to the variance of the images to obtain 
an estimated sub-space for each facial property, 

d. concatenating the sub-spaces to provide a single over-exhaustive space, 

e. approximating each image of the first and second series on the over- 
exhaustive space to obtain approximated versions of each image on each estimated 
property subspace, 

f. generating overall approximated versions of each image on the whole over- 
exhaustive space, 

g. comparing the overall approximated version of each image with the initial 
image to determine an error value for each image, 

h. sub-dividing the error value for each image into a sub-error for each estimated 
property sub-space in proportion to the variance of that sub-space, 

i. combining each sub-error for each image with the approximated version of 
that image on the estimated property sub-space, to obtain a new approximated version 
in the property sub-space for each image, 

j. coding the new approximated versions of the images according to their 
variance to obtain new estimated sub-spaces. 
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3. A method of determining face sub-spaces according to claim 2, further 
comprising approximating each image on the new estimated sub-spaces as described 
in steps 'a' to 'j' and then repeating steps 'd' to ( j' until the sub-spaces have 
stabilised. 

4. A method of determining face sub-spaces according to claim 2 or claim 3, 
wherein three or more series of images are generated, a different predetermined facial 
property being modified in each series. 

5. A method according to claim 4, wherein the predetermined facial properties 
are categorised as at least some of identity, expression, pose, lighting and age. 

6. A method according to any of claims 2 to 5, wherein at least one further series 
of images is generated, a further predetermined facial property being modified in the 
series. 

7. A method of determining face sub-spaces substantially as hereinbefore 
described. 
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1 . This report has been drawn on the basis of (substitute sheets which have been furnished to the receiving Office in 
response to an invitation under Article 14 are referred to in this report as "originally filed" and are not annexed to 
the report since they do not contain amendments (Rules 70. 16 and 70. 1 7).): 

Description, pages: 

1 -3,5-7,9 as originally filed 

4,8,10-12 as received on 09/01/2001 with letter of 03/01/2001 
Claims, No.: 

1-7 as received on 09/01/2001 with letter of 03/01/2001 
Drawings, sheets: 

1/4-4/4 as originally filed 

2. With regard to the language, all the elements marked above were available or furnished to this Authority in the 
language in which the international application was filed, unless otherwise indicated under this item. 

These elements were available or furnished to this Authority in the following language: , which is: 

□ the language of a translation furnished for the purposes of the international search (under Rule 23.1 (b)). 

□ the language of publication of the international application (under Rule 48.3(b)). 

□ the language of a translation furnished for the purposes of international preliminary examination (under Rule 
55.2 and/or 55.3). 

3. With regard to any nucleotide and/or amino acid sequence disclosed in the international application, the 
international preliminary examination was carried out on the basis of the sequence listing: 

□ contained in the international application in written form. 

□ filed together with the international application in computer readable form. 

□ furnished subsequently to this Authority in written form. 

□ furnished subsequently to this Authority in computer readable form. 

□ The statement that the subsequently furnished written sequence listing does not go beyond the disclosure in 
the international application as filed has been furnished. 

□ The statement that the information recorded in computer readable form is identical to the written sequence 
listing has been furnished. 

4. The amendments have resulted in the cancellation of: 
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□ 



the description, 
the claims, 



pages: 



□ 



Nos.: 



□ 



the drawings, 



sheets: 



5. □ This report has been established as if (some of) the amendments had not been made, since they have been 

considered to go beyond the disclosure as filed (Rule 70.2(c)): 

(Any replacement sheet containing such amendments must be referred to under item 1 and annexed to this 
report.) 

6. Additional observations, if necessary: 

III. Non-establishment of opinion with regard to novelty, inventive step and industrial applicability 

1. The questions whether the claimed invention appears to be novel, to involve an inventive step (to be non- 
obvious), or to be industrially applicable have not been examined in respect of: 

□ the entire international application. 
H claims Nos. 6, 7. 



□ the said international application, or the said claims Nos. relate to the following subject matter which does 
not require an international preliminary examination (specify): 

H the description, claims or drawings (indicate particular elements below/) or said claims Nos. 6, 7 are so 
unclear that no meaningful opinion could be formed (specify): 
see separate sheet 

□ the claims, or said claims Nos. are so inadequately supported by the description that no meaningful opinion 
could be formed. 

□ no international search report has been established for the said claims Nos. . 

2. A meaningful international preliminary examination report cannot be carried out due to the failure of the nucleotide 
and/or amino acid sequence listing to comply with the standard provided for in Annex C of the Administrative 



□ the written form has not been furnished or does not comply with the standard. 

□ the computer readable form has not been furnished or does not comply with the standard. 

V. Reasoned statement under Article 35(2) with regard to novelty, inventive step or industrial applicability; 
citations and explanations supporting such statement 



because: 



Instructions: 
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1. Statement 



Novelty (N) 



Yes: 
No: 



Claims 
Claims 



1 



■5 



Inventive step (IS) 



Yes: 
No: 



Claims 
Claims 



1 



■5 



Industrial applicability (IA) Yes: Claims 1 -5 

No: Claims 

2. Citations and explanations 
see separate sheet 

VII. Certain defects in the international application 

The following defects in the form or contents of the international application have been noted: 
see separate sheet 

VIII. Certain observations on the international application 

The following observations on the clarity of the claims, description, and drawings or on the question whether the 
claims are fully supported by the description, are made: 
see separate sheet 
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Re Item III 

Non-establishment of opinion with regard to novelty, inventive step and industrial 
applicability 

1. Independent claim 6 does not meet the requirements of Article 6 and Rule 6.3(a) 
PCT in that it does not clearly define the scope of the monopoly for which protection 
is sought since said claim is expressed in vague and ambiguous terms. 

The vague, functional and excessively concise wording of this claim does not allow 
the skilled reader to even only guess how the recited iterative algorithm is supposed 
to "maximise the probability of coding across these non-orthogonal sub-spaces". 
Independent claim 7 recites a method of determining face sub-spaces defined only 
by a completely undefined reference to the description in its entirety. 
The wording of both claims 6 and 7 is unclear (Article 6 PCT) to such an extent that 
it appears impossible to even only attempt an assessment of novelty and inventive 
step for such a broadly formulated claim. 

Re Item V 

Reasoned statement under Article 35(2) with regard to novelty, inventive step or 
industrial applicability; citations and explanations supporting such statement 

2. The documents cited in the International Search Report, all cited in the application, 
only indicate that it is in general known to use autocorrelation techniques in the field 
of human face recognition for determining eigenvectors (e.g. eigenfaces, 
eigensignatures) of a learning set of face images. These eigenvectors are 
determined for series of images in which only a chosen visual feature varies, e.g. one 
series for different illumination conditions, one series for different poses of one or 
more identical faces, etc. The eigenvectors can be thus grouped in sets that define 
facial sub-spaces, each sub-space associated to a given visual feature (e.g. pose, 
or facial expression). 

However, there is the problem that these sub-spaces are in fact not perfectly 
orthogonal. In other words, not only the desired visual feature is coded by the 
eigenvectors of a given sub-space, but also an unknown mixture of the other types 
of visual features, depending on the initial training series of face images. 
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3. The present invention discloses a method as set out in claim 1 for determining sub- 
spaces of (human) facial variations. 

Initial series of face images, each series associated to a given facial property (e.g. 
pose, illumination, expression, etc.), are analysed, and a plurality of eigenvectors 
($®) (and eigenvalues (A a) )) determined for each series, thereby determining an 
estimated sub-space for each facial property (claim 1 , steps "a." to "a"). 
An over-exhaustive space is then determined (claim 1, step "d.") by concatenating 
said eigenvectors (matrix M, cf. page 8, line 9, page 9, line 3). 
By re-coding each face image {q) of the initial series on the basis of the this over- 
exhaustive space, approximated versions (q)) thereof are generated for each 
estimated facial property sub-space, and an error value (r) between each such 
approximated version and the corresponding initial image can be determined (claim 
1, steps "e." to "g."). 

This error value is split into sub-error values, each sub-error value associated to a 
given estimated facial property sub-space and being proportional to the variance of 
that sub-space (claim 1 , step "h."). 

For each estimated facial property sub-space, a new approximated version of each 
face image of the initial series is obtained by combining the initial approximated 
version and these sub-error values (claim 1 , step "i."). 

These new approximated versions can be again analysed (determining updated 
eigenvectors), thereby determining a new estimated sub-space for each facial 
property with improved orthogonality (claim 1 , step "j."). 

The method can be iterated (from steps "d." to "j.", cf. claim 2), thereby determining 
a sequence of newly estimated sub-spaces for each facial property, the sub-spaces 
being progressively more stable and having greater orthogonality with respect to 
each other. 

Consequently, the subject-matter set out in present claims 1 to 5, and particularly in 
claim 1 , is considered to be novel and non-obvious with respect to the disclosures of 
the available prior art. It is also evident that the invention is industrially applicable. 
The requirements of paragraphs (1) to (4) of Article 33 PCT are thus met. 
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Re Item VII 

Certain defects in the international application 

4. None of the present claims is provided with reference signs placed in parentheses 
(Rule 6.2(b) PCT), which in the present case would have been appropriate for 
indicating the necessary correspondence between claimed and described features. 

Re Item VIII 

Certain observations on the international application 

5. The application does not meet the requirements of Article 6 PCT, because 
independent claims 1 and 6 are not clear. 

6. Present independent claims 1 and 6 have only partially overlapping scope, and it is 
not clear from their wording how the claimed features are supposed to correspond 
to each other. 

It does not seem that such a multiplicity of independent claims is strictly necessary 
and hence the number would not appear to be reasonable (Rule 6.1 (a) PCT). 
The claims should have contained only the minimum number of independent claims 
necessary to clearly and sufficiently define the invention, with dependent claims as 
appropriate, Article 6 and Rule 6.4 PCT. 

In the present case, it would have been considered appropriate to use only one 
independent claim per category. 

7. Furthermore, independent claim 6 considered in isolation does not meet the 
requirements of Article 6 and Rule 6.3 (a) PCT in that it does not clearly define the 
scope of the monopoly for which protection is sought since said claim is expressed 
in vague and ambiguous terms. 

The vague, functional and excessively concise wording of this claim does not allow 
the skilled reader to even only guess how the recited iterative algorithm is supposed 
to "maximise the probability of coding across these non-orthogonal sub-spaces". 
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